The prior art techniques of this kind are used in automatic doors employing photosensors, footboards, etc., warning devices for informing a person of entry or intrusion, and metal sensors. Any of these devices makes use of a noncontact sensor, such as a photosensor, microswitch, electrostatic field-type proximity switch, or electromagnetic sensor, or a mechanical switch, and detects opening or closure of an electrical contact, making or breaking of an electromagnetic wave path, a change in an electric field, or a change in a magnetic field which is caused when an object or a person makes contact with, approaches, or passes through, the device to turn on or off a desired electrical device, such as a buzzer, meter, automatic door, relay, monitor television , or an electrically controlled machine.
This electrical device cannot be controlled, e.g., turned on and off, unless an object or person is close to the device and makes relatively large movement. Since a change in the state of a minute portion of an object or human body cannot be detected by a sensor, an input device consisting principally of keyswitches has been heretofore most frequently used to energize various electrical devices. As an example, various electrical devices are installed on an automobile, and various keyswitches, volumes, etc. are disposed corresponding to those electrical devices. However, if the driver stretches his or her arm or twists around to operate a switch or volume control, then the driving is endangered. Also, it is easy to meticulously operate a switch or volume control, because the driver cannot keep his or her eyes off the front view for a relatively long time to watch a device. Accordingly, it may be contemplated to install a speech recognition apparatus which recognizes the driver's speech and controls various electrical devices. Unfortunately, a large amount of noise takes place inside the automobile and so the recognition involves noticeable error.
In order to automatically control or energize various electrical devices according to the change in the state of a small portion within a broad region and to permit the driver to control various electrical devices relatively precisely in a noncontact manner without requiring great care or large motion, the present inventor has developed an apparatus that turns on and off devices installed on a vehicle in response to the motion of driver's eyes and mouth, as disclosed in Japanese Patent application No. 272793/1985.
This apparatus makes use of image pattern recognition techniques. In particular, this apparatus uses a camera means for converting an image, or information in the form of light, into an electrical signal and a position-detecting means that detects the position of certain portions of the image. In operation, the apparatus takes a picture of an object or person, such as an automobile driver, and detects the positions of the certain portions of the picture, such as the driver's eyes and mouth.
Since the brightness inside the automobile varies, an illuminating means for illuminating the driver, a brightness-setting means for setting the brightness of the illuminating means, and a brightness control means are provided. The brightness control means detects the brightness on the driver's face and adjusts the setting of the brightness-setting means to change the brightness. Thus, the brightness on the driver's face is maintained constant to prevent the image processing from producing error due to variations in the brightness.
The position of the driver's face may be changed by vibration of the automobile body or may vary because of his or her unintentional minute motion or a change in the posture. Also, the eyes and mouth may be intentionally moved to control electrical devices in a noncontact manner as described later. To precisely extract information about the eyes and mouth from image information in response to the changes in the positions of the face, eyes, and mouth, the apparatus further includes a storage means for storing the detected positions, a window setting means for setting a region narrower than the image produced by the camera means according to the stored positions, a means for setting the region covered by a position-detecting means to the narrower region after a certain period of time elapses since the detected positions are stored in the storage means, and an updating means for updating the positions of the aforementioned certain portions within the narrower region which are stored in the storage means. Once the positions of the certain portions, i.e., the eyes and mouth, are detected, the scan made to detect the eyes and mouth is limited to the narrower region and so they can be detected quickly. Further, the accuracy with which the detection is made is enhanced. Consequently, the apparatus follows the eyes and mouth quickly and precisely.
This apparatus is further equipped with a state change-detecting means for detecting the states of the eyes and mouth at successive instants of time to detect the changes in the states. Also, the apparatus includes an output-setting means which supplies a control signal or electric power to an electrical device according to the changes in the states. Specifically, when the states of the monitored eyes and mouth are found to change in a predetermined manner, i.e, it is ready to activate the electrical device, electric power is supplied to the device according to the change.
The apparatus enables the driver to control the electrical device by moving his or her eyes or mouth while assuming a posture adequate to drive the automobile. Therefore, the electrical device installed on the automobile can be quite easily operated. This contributes to a comfortable and safe drive. As an example, when the driver utters a word to indicate something, the electrical device is controlled according to the shape of the mouth. If the driver utters no word but moves the mouth intentionally as if to utter a word, then the electrical device is controlled according to the shape of the mouth. Since the operation of the device is not affected by utterance, the detection involves no error in spite of noise produced inside the passenger's compartment. Also, if the radio set is played, or if a passenger is speaking loudly, it is unlikely that the electrical device is caused to malfunction.
The concept of the aforementioned apparatus can be similarly applied to the case where a person other than an automobile driver is monitored. For example, a similar apparatus allows a patient with an advanced disease to operate, stop, or control the surrounding medical instruments or assisting instruments with his or her eyes and mouth.
The apparatus can also monitor a machine to detect abnormality and protect the machine. A certain part or portion of the machine is checked for trouble. If this part or portion operates abnormally, the operation of the machine is stopped, or a warning device is operated. In this way, the above described apparatus can be also employed with similar utility to monitor an object other than a person.
Further, the invention can be utilized to monitor a broad region such as a natural sight, especially to monitor animals or vehicles moving in the region. For instance, a gate in a safari park can be opened and closed according to the movement of a vehicle or fierce animals. For a manufacturing plant a belt conveyor line can be monitored to check the parts or products on the conveyor. When they move in a given direction, a safety device is operated, or equipment for the next manufacturing step is run. In this way, the aforementioned apparatus can be used with similar utility in the same manner as the foregoing.
The apparatus described above can detect the driver's head, face, and pupils with high accuracy and yield the foregoing advantages when the monitored object, such as the face of the automobile driver, has a relatively uniform brightness, typically encountered when no car is running in the opposite direction at night and substantially only the interior light illuminates the face, thus permitting the monitoring. However, when the driver's face or head is illuminated with intense light emanating from the headlamps either on a car running in the opposite direction or on a succeeding car even at night, or when the sunlight is intense in the daytime, the external light stronger than the light emitted from the interior light is reflected or intercepted by the driver's face or head. In this situation the brightness on the face frequently becomes nonuniform. That is, intense light is reflected from only a portion of the face; the remaining portion is in shadow and darker. As an example, when the automobile is running in fine weather under the sun located to the right of the automobile, the surroundings of the right eye are very bright, while the surroundings of the left eye are quite dark. In this nonuniform illumination, the accuracy with which the driver's pupils are detected deteriorates, because the apparatus uses only one threshold value in digitizing the whole obtained image. Also, the shape of the driver's mouth is detected with decreased accuracy.
Accordingly, the present inventor has developed an improvement over the aforementioned known apparatus to detect elements, such as the pupils or the mouth or both, of a monitored object, such as the driver's face, with increased accuracy, as disclosed in Japanese Patent application No. 169325/1987. The improved apparatus arithmetically obtains a first gradation histogram for each of small neighboring regions, for example the right half and the left half, within a desired portion such as a human face included in the monitored image. Then, a threshold value for each region is determined, based on the histogram. Information about the gradation of the image is digitized, and a characteristic index (HTY) which indicates the boundary between the hair and the forehead, for example, is determined. This boundary extends through the neighboring regions on the monitored face. Opposite sides of the boundary differ in gray level. A second gradation histogram is created from information about the gradation of an image of a set region S.sub.d based on the determined characteristic index (HTY). The set region S.sub.d contains the eyes. Then, a threshold value (TH.sub.e) is determined according to this histogram to digitize the gradation of the image of the region (S.sub.d). Thus, the positions of certain small portion or portions, such as pupils, within the region (S.sub.d) are detected. The certain small portion can be a mouth instead of pupils.
Determination of a threshold value from a gradation histogram and digitization an analog signal are known in the field of object recognition image processing. These techniques are adequate to separate an object located in front of the background from the background of the image when the concentration of the image varies. Accordingly, this improved apparatus can precisely detect the characteristic index which indicates the upper end of the forehead. This digitization is adequate to detect a characteristic index (HTY) indicating the boundary between the background, or hair, and the main portion, or forehead, in each divided region even if the monitored object is not uniformly illuminated or the brightness of the light source itself varies. Hence, the index (HTY) can be detected with accuracy. The index (HTY) represents a reference position on the detected object, or face.
The region (S.sub.d) surrounding the eyes is set according to the characteristic index (HTY). A threshold value is set according to a gradation histogram obtained from this region (S.sub.d). Then, an analog signal is transformed into binary codes, using the threshold value. These techniques are adequate to define the given region (S.sub.d) containing the certain small regions, or pupils, of the detected object, and to separate the pupils whose gray levels suddenly change in the region (S.sub.d), from the background, or the surroundings of the pupils if the object is illuminated asymmetrically or the brightness of the light source itself varies. Consequently, the certain small portions, or the pupils, can be detected accurately. Also, the small portions can be mouth or lips.
In this manner, the improved apparatus is capable of detecting given portions of an object accurately if the object is illuminated asymmetrically or the brightness of the light source itself varies.
If the driver sitting on the driver's seat of an automobile shifts the seat forward or rearward to adjust the posture for driving, the distance between the camera means and the subject, or face, changes. At this time, an automatic focusing device prevents the obtained image from getting blurred. However, the possibility that elements of the image are incorrectly detected, e.g., the nostrils are regarded as the mouth, increases.